Methods and systems for implementing customized motions based on individual profiles for identified users

ABSTRACT

A teleoperation system includes a robot comprising an actuator configured to move at least a portion of the robot, and a remote computing device comprising: one or more processors, one or more sensors communicatively coupled to the one or more processors, a non-transitory memory component communicatively coupled to the one or more processors, and machine readable instructions stored in the non-transitory memory component. The remote computing device obtains information about a user proximate to the remote computing device, identifies the user based on the obtained information, obtains an action of a user, retrieves an individual profile for the user based on the identified user, determines an intended instruction related to a task based on the action of the user related to the task and the individual profile for the user, and instructs the robot to implement the task with the actuator based on the intended instruction.

TECHNICAL FIELD

The present disclosure generally relates to methods and systems forcontrolling a robot to implement customized motions, and, moreparticularly to interpreting intended instructions of a user based on anindividual profile for the user and implementing customized motionsbased on the intended instructions.

BACKGROUND

Robots may implement tasks on behalf of users. Robots may receiveinstructions from users and operate according to the receivedinstructions. For example, a robot may move to a certain location inresponse to receiving an instruction for moving to the location from auser. As another example, a robot may move its arm in response toreceiving an instruction for moving the arm from a user. A robot mayinteract with many users. Different users may have different intentsand/or styles regarding the operations of the robot, e.g., a movingspeed, an amount of force that the robot applies, a degree of accuracyin conducting a task, etc.

Accordingly, a need exists for methods and systems for interpreting theintention of a user interacting with a robot and implementing customizedmotions based on an individual profile for the user.

SUMMARY

In one aspect, a teleoperation system includes a robot including anactuator configured to move at least a portion of the robot; and aremote computing device. The remote computing device includes one ormore processors, one or more sensors communicatively coupled to the oneor more processors, a non-transitory memory component communicativelycoupled to the one or more processors, and machine readable instructionsstored in the non-transitory memory component. The remote computingsystem obtains information about a user proximate to the remotecomputing system with the one or more sensors, identifies the user basedon the obtained information, obtains an action of the user, retrieves anindividual profile for the user based on the identified user; determinean intended instruction related to a task based on the action of theuser related to the task and the individual profile for the user, andinstructs the robot to implement the task based on the intendedinstruction. The individual profile includes intent parameters relatedto the action of the user. In some embodiments, the remote computingdevice may include an output device configured to provide feedback tothe user based on an operation of the device, and the remote computingdevice determines the feedback based on the individual profile for theuser. In some embodiments, the remote computing device comprises adisplay configured to display a view related to the robot, and theremote computing device determines a type of the view based on theindividual profile for the user.

In another aspect, a robot system is provided. The robot system includesone or more processors, one or more sensors communicatively coupled tothe one or more processors, a non-transitory memory componentcommunicatively coupled to the one or more processors, and machinereadable instructions stored in the non-transitory memory component. Therobot system obtains information about a user proximate to the robotsystem with the one or more sensors, identifies the user based on theobtained information, obtains an action of a user, retrieves anindividual profile for the user based on the identified user, determinesan intended instruction related to a task based on the action of theuser related to the task and the individual profile for the user, andimplements the task based on the intended instruction. The individualprofile includes intent parameters related to the action of the user.

In yet another aspect, a method for operating a robot is provided. Themethod includes obtaining, with one or more sensors of a remotecomputing device, information about a user proximate to the remotecomputing device, identifying, by a controller of the remote computingdevice, the user based on the obtained information, obtaining, by thecontroller of the remote computing device, an action of the user,retrieving, by the controller of the remote computing device, anindividual profile for the user based on the identified user,determining, by the controller of the remote computing device, anintended instruction related to a task based on the action of the userrelated to the task and the individual profile for the user, andimplementing, by the controller of the remote computing device, the taskbased on the intended instruction. The individual profile includesintent parameters related to the action of the user.

These and additional features provided by the embodiments describedherein will be more fully understood in view of the following detaileddescription, in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments set forth in the drawings are illustrative and exemplaryin nature and not intended to limit the subject matter defined by theclaims. The following detailed description of the illustrativeembodiments can be understood when read in conjunction with thefollowing drawings, where like structure is indicated with likereference numerals and in which:

FIG. 1A schematically depicts a robot interacting with users, accordingto one or more embodiments described and illustrated herein;

FIG. 1B schematically depicts a robot interacting with a remote user,according to one or more embodiments described and illustrated herein;

FIG. 2 schematically depicts a robot, according to one or moreembodiments described and illustrated herein;

FIG. 3 schematically depicts a flowchart of a method of implementing atask required by a user, according to one or more embodiments describedand illustrated herein;

FIG. 4A depicts a robot operating based on a workspace scaling value fora user, according to one or more embodiments described and illustratedherein;

FIG. 4B depicts a robot operating based on a workspace scaling value foranother user, according to one or more embodiments described andillustrated herein;

FIG. 5A depicts a robot interpreting an intended instruction of a userand operating based on the intended instruction, according to one ormore embodiments described and illustrated herein; and

FIG. 5B depicts a robot interpreting an intended instruction of anotheruser and operating based on the intended instruction, according to oneor more embodiments described and illustrated herein.

DETAILED DESCRIPTION

The embodiments described herein are directed to methods and systems forcontrolling a robot to implement customized motions. A robot may receiveoperation instructions from various users, and the instructions fromvarious users need to be interpreted differently based on personalintents and/or styles of the users. Thus, the robot needs to interpretthe instructions from the user to implement customized operations.Robots according to the present disclosure address the problems ofconventional robots by identifying a user and interpreting the intentionof the user based on an individual profile for the identified user.

The robot system includes one or more processors, one or more sensorscommunicatively coupled to the one or more processors, a non-transitorymemory component communicatively coupled to the one or more processors,and machine readable instructions stored in the non-transitory memorycomponent. The robot obtains information about a user proximate to therobot with the one or more sensors, identifies the user based on theobtained information, obtains an action of the user, retrieves anindividual profile for the user based on the identified user; determinean intended instruction related to a task based on the action of theuser related to the task and the individual profile for the user, andimplements the task based on the intended instruction. The individualprofile includes intent parameters related to the action of the user.

Referring now to FIG. 1A, a robot 100 interacting with one or more usersis illustrated. For example, the robot 100 is interacting with user A120 and user B 130. The robot 100, which is illustrated generically inFIG. 1A, may take on any size and configuration. For example, the robot100 may be configured as service robot to assist humans in the home,workplace, school, or healthcare facility, such as the robot 100illustrated in FIG. 1A. In another embodiment, the robot may be aproduction robot utilized within a manufacturing facility. It should beunderstood that the embodiments described herein are not limited to anytype of robot.

The robot 100 illustrated in FIG. 1A generally comprises image capturingdevices 102 a, 102 b, an arm 106, a gripping assembly 108, a locomotiondevice 104, a screen 110, and a microphone 112. The image capturingdevices 102 a, 102 b may be configured as digital cameras capable ofacquiring still image and/or digital video. In an alternativeembodiment, the robot 100 may be equipped with only one image capturingdevice to more than two image capturing devices. The image capturingdevices 102 a, 102 b depicted in FIG. 1A may enable the robot 100 todetect users, and recognize users, as described below.

The locomotion device 104 is utilized by the robot 100 to maneuverwithin an operating space 101. In the embodiment depicted in FIG. 1A,the locomotion device is tracked locomotion device. However, in otherembodiments, the robot 100 may include one or more locomotive devicesother than a tracked locomotive device. For example, the robot 100 maymaneuver within the operating space 101 using one or more wheels orlegs. In some embodiments, the robot 100 may include more than twolocomotive devices. In some embodiments, the robot 100 may be anunmanned aerial vehicle, or an unmanned submersible.

The arm 106 and gripping assembly 108 may be servo-actuated in oneembodiment to manipulate objects that the robot 100 encounters withinthe operating space. Other actuation mechanisms may be utilized, such asby pneumatic drives, hydraulic drives, electro-active polymer motors,etc. In some embodiments, the robot 100 may include only one arm andgripping assembly or more than two arms and gripping assemblies.

The screen 110 may display images, videos, texts, etc. which are visibleto users proximate to the robot 100. For example, the screen 110 maydisplay texts that describe an operation that the robot 100 is currentlyimplementing, e.g., picking up a water bottle. As another example, thescreen 110 may display the picture of a user that the robot 100 iscurrently interacting with. The microphone 112 may record audio externalto the robot 100, e.g., voice by user A 120 or user B 130.

In some embodiments, a user may interact with the robot 100 by providinginstructions using wearable devices. For example, user A 120 may wear awearable device 122 that is communicatively coupled to the robot 100.The wearable device 122 may be any type of wearable device, e.g., asmart wrist band, a smart watch, a smart glove, etc. The wearable device122 obtains the motion data of user A 120 and transmits the motion datato the robot 100, which will be described in detail with reference toFIG. 2 below. The robot 100 may operate based on the received motiondata from user A 120. In some embodiments, user A 120 may be remotelylocated from the robot 100.

In some embodiments, a user may interact with the robot 100 using a userinterface. For example, user B 130 holds a user interface 132 that iscommunicatively coupled to the robot 100. The user interface 132 may bean input device configured to receive instructions from a user foroperating the robot 100. For example, the input device may be a joystickthat the user can manipulate to operate the robot 100. The user maycontrol the moving direction and speed of the robot 100 by controllingthe joystick. The user interface 132 may be communicatively coupled tothe robot 100 in order to send instructions to the robot 100, which willbe described below with reference to FIG. 2.

FIG. 1B depicts a robot 100 interacting with a remote user via a remotecomputing device 140. For example, the robot 100 is interacting withuser C 150 who is located at a remote location from the robot 100. Therobot 100 may communicate with the remote computing device 140 throughwireless communication which will be described in detail with referenceto FIG. 2. The remote computing device 140 may include one or more imagecapturing devices 142, a screen 144, a microphone 146, a user interface148, and a speaker 149.

The one or more image capturing devices 142 may be configured as digitalcameras capable of acquiring still image and/or digital video. The oneor more image capturing devices 142 depicted in FIG. 1B may enable therobot 100 to detect users, and recognize users, as described below.

The screen 144 may display images, videos, texts, etc. which are visibleto users proximate to the robot 100. In embodiments, the screen 144 maydisplay views related to the robot 100. For example, the screen 144 maydisplay the overhead view of the robot 100 such that the user C 150 maysee the robot 100 and its surrounding. As another, the screen 144 maydisplay the view of the image capturing devices 102 a and 102 b of therobot 100 such that the user C 150 may view what the robot 100 iscurrently viewing. The type of view (e.g., an overhead view, a robot'sdirect view, etc.) may be determined based on an individual profile forthe user C 150.

The microphone 146 may record audio external to the remote computingdevice 140, e.g., voice by user C 150. In some embodiments, a user mayinteract with the robot 100 using a user interface 148. The userinterface 148 may be an input device configured to receive instructionsfrom a user for operating the robot 100. For example, the input devicemay be a joystick that the user can manipulate to operate the robot 100.The user may control the moving direction and speed of the robot 100 bycontrolling the joystick. In response to the operation on the userinterface 148, the remote computing device 140 sends instructions to therobot 100, which will be described below with reference to FIG. 2.

In some embodiments, the user C 150 may wear a virtual reality device(e.g., a virtual reality headset) that is communicatively coupled to theremote computing device 140. The virtual reality device may providehaptic feedback to the user C 150 based on instructions from the remotecomputing device 140. For example, when the robot 100 contacts anobstacle, the robot 100 transmits a signal indicating the contact to theremote computing device 140. The remote computing device 140 instructsthe virtual reality device to vibrate in response to receiving thesignal indicating the contact.

Referring now to FIG. 2, various internal components of the robot 100are illustrated. The robot 100 includes a controller 210 that includesone or more processors 202 and one or more memory modules 204, the imagecapturing devices 102 a, 102 b, a satellite antenna 220, actuator drivehardware 230, network interface hardware 240, the screen 110, themicrophone 112, and a speaker 114. In some embodiments, the one or moreprocessors 202, and the one or more memory modules 204 may be providedin a single integrated circuit (e.g., a system on a chip). In someembodiments, the one or more processors 202, and the one or more memorymodules 204 may be provided as separate integrated circuits.

Each of the one or more processors 202 is configured to communicate withelectrically coupled components, and may be configured as anycommercially available or customized processor suitable for theparticular applications that the robot 100 is designed to operate. Eachof the one or more processors 202 may be any device capable of executingmachine readable instructions. Accordingly, each of the one or moreprocessors 202 may be a controller, an integrated circuit, a microchip,a computer, or any other computing device. The one or more processors202 are coupled to a communication path 206 that provides signalinterconnectivity between various modules of the robot 100. Thecommunication path 206 may communicatively couple any number ofprocessors with one another, and allow the modules coupled to thecommunication path 206 to operate in a distributed computingenvironment. Specifically, each of the modules may operate as a nodethat may send and/or receive data. As used herein, the term“communicatively coupled” means that coupled components are capable ofexchanging data signals with one another such as, for example,electrical signals via conductive medium, electromagnetic signals viaair, optical signals via optical waveguides, and the like.

Accordingly, the communication path 206 may be formed from any mediumthat is capable of transmitting a signal such as, for example,conductive wires, conductive traces, optical waveguides, or the like.Moreover, the communication path 206 may be formed from a combination ofmediums capable of transmitting signals. In one embodiment, thecommunication path 206 comprises a combination of conductive traces,conductive wires, connectors, and buses that cooperate to permit thetransmission of electrical data signals to components such asprocessors, memories, sensors, input devices, output devices, andcommunication devices. Additionally, it is noted that the term “signal”means a waveform (e.g., electrical, optical, magnetic, mechanical orelectromagnetic), such as DC, AC, sinusoidal-wave, triangular-wave,square-wave, vibration, and the like, capable of traveling through amedium.

The one or more memory modules 204 may be coupled to the communicationpath 206. The one or more memory modules 204 may include a volatileand/or nonvolatile computer-readable storage medium, such as RAM, ROM,flash memories, hard drives, or any medium capable of storing machinereadable instructions such that the machine readable instructions can beaccessed by the one or more processors 202. The machine readableinstructions may comprise logic or algorithm(s) written in anyprogramming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or5GL) such as, for example, machine language that may be directlyexecuted by the processor, or assembly language, user-orientedprogramming (OOP), scripting languages, microcode, etc., that may becompiled or assembled into machine readable instructions and stored onthe one or more memory modules 204. Alternatively, the machine readableinstructions may be written in a hardware description language (HDL),such as logic implemented via either a field-programmable gate array(FPGA) configuration or an application-specific integrated circuit(ASIC), or their equivalents. Accordingly, the methods described hereinmay be implemented in any conventional computer programming language, aspre-programmed hardware elements, or as a combination of hardware andsoftware components.

The one or more memory modules 204 may be configured to store one ormore modules, each of which includes the set of instructions that, whenexecuted by the one or more processors 202, cause the robot 100 to carryout the functionality of the module described herein. For example, theone or more memory modules 204 may be configured to store a robotoperating module, including, but not limited to, the set of instructionsthat, when executed by the one or more processors 202, cause the robot100 to carry out general robot operations. Furthermore, the one or morememory modules 204 may be configured to store a face recognition module,an action recognition module, and individual profiles for users, thedetails of which is described below with reference to FIG. 3. It shouldbe understood that in some embodiments, the one or more memory modules204 may be configured to store only a subset of the face recognitionmodule, the action recognition module, and individual profiles forusers. Other data may be stored in the one or more memory modules 204 toprovide support for functionalities described herein.

The image capturing devices 102 a, 102 b may be coupled to thecommunication path 206. The image capturing devices 102 a, 102 b mayreceive control signals from the one or more processors 202 to acquireimage data of a surrounding operating space, and to send the acquiredimage data to the one or more processors 202 and/or the one or morememory modules 204 for processing and/or storage. The image capturingdevices 102 a, 102 b may be directly connected to the one or more memorymodules 204, or, in an alternative embodiment, include dedicated memorydevices (e.g., flash memory) that are accessible to the one or moreprocessors 202 for retrieval.

Each of the image capturing devices 102 a, 102 b may have any suitableresolution and may be configured to detect radiation in any desirablewavelength band, such as an ultraviolet wavelength band, anear-ultraviolet wavelength band, a visible light wavelength band, anear infrared wavelength band, or an infrared wavelength band. In someembodiments, at least one of the image capturing devices 102 a, 102 bmay be a standard definition (e.g., 640 pixels×480 pixels) camera. Insome embodiments, at least one of the image capturing devices 102 a, 102b may be a high definition camera (e.g., 1440 pixels×1024 pixels or 1280pixels×1024 pixels). In some embodiments, at least one of the imagecapturing devices 102 a, 102 b may have a resolution other than 640pixels×480 pixels, 1440 pixels×1024 pixels, or 1280 pixels×1024 pixels.The image capturing devices 102 a, 102 b may provide image data in theform of digital video and/or one or more digital photographs.

The robot 100 includes a satellite antenna 220 coupled to thecommunication path 206 such that the communication path 206communicatively couples the satellite antenna 220 to other modules ofthe robot 100. The satellite antenna 220 is configured to receivesignals from global positioning system satellites. Specifically, in oneembodiment, the satellite antenna 220 includes one or more conductiveelements that interact with electromagnetic signals transmitted byglobal positioning system satellites. The received signal is transformedinto a data signal indicative of the location (e.g., latitude andlongitude) of the satellite antenna 220 or a user positioned near thesatellite antenna 220, by the one or more processors 202. In someembodiments, the robot 100 may not include the satellite antenna 220.

The actuator drive hardware 230 may comprise the actuators andassociated drive electronics to control the locomotion device 104, thearm 106, the gripping assembly 108, and any other external componentsthat may be present in the robot 100. The actuator drive hardware 230may be configured to receive control signals from the one or moreprocessors 202 and to operate the robot 100 accordingly. The operatingparameters and/or gains for the actuator drive hardware 230 may bestored in the one or more memory modules 204. The operating parametersand/or gains for the actuator drive hardware 230 may be adjusted basedon individual profiles stored in the one or more memory modules 204. Forexample, one or more drive gain parameters for controlling thelocomotion device 104 may be adjusted according to different drive gainparameters in the individual profiles. Specifically, when user A 120 isidentified by the robot 100, drive gain parameters in the individualprofile for user A are retrieved, and the one or more drive gainparameters for controlling the locomotion device 104 may be adjustedaccording to the retrieved drive gain parameters. As another example, adrive gain parameter, a torque, and/or any other parameters forcontrolling the arm 106 and the gripping assembly 108 may be adjustedaccording to different drive gain parameters, torques, and/or any otherparameters in the individual profiles. Specifically, when user B 130 isidentified by the robot 100, drive gain parameters, torques, and/or anyother parameters in the individual profile for user B are retrieved, andthe drive gain parameters, torques, and/or any other parameters forcontrolling the arm 106 and the gripping assembly 108 may be adjustedaccording to the retrieved drive gain parameters, torques, and/or anyother parameters.

The robot 100 includes the network interface hardware 240 forcommunicatively coupling the robot 100 with a mobile device 250, aremote computing device 140, the wearable device 122, and the userinterface 132. The mobile device 250 or the remote computing device 140may authenticate itself before it initiates communication with the robot100 through the network interface hardware 240. The display of themobile device 250 or the remote computing device 140 may displayinformation to the user of the mobile device 250 or the remote computingdevice, e.g., the view that the robot is seeing with its cameras, a mapof a room or building that the robot is in, the path of the robot, or ahighlight of an object to be grasped. The network interface hardware 240may be coupled to the communication path 206 and may be configured as awireless communications circuit such that the robot 100 may communicatewith external systems and devices. The network interface hardware 240may include a communication transceiver for sending and/or receivingdata according to any wireless communication standard. For example, thenetwork interface hardware 240 may include a chipset (e.g., antenna,processors, machine readable instructions, etc.) to communicate overwireless computer networks such as, for example, wireless fidelity(Wi-Fi), WiMax, Bluetooth, IrDA, Wireless USB, Z-Wave, ZigBee, or thelike. In some embodiments, the network interface hardware 240 includes aBluetooth transceiver that enables the robot 100 to exchange informationwith the mobile device 250 (e.g., a smartphone) via Bluetoothcommunication. In some embodiments, the robot 100 may not include thenetwork interface hardware 240.

The network interface hardware 240 may receive motion data from one ormore motion sensors 280 of the wearable device 122. The one or moremotion sensors 280 may include inertial measurement units. Each of theone or more motion sensors 280 may include one or more accelerometersand one or more gyroscopes. Each of the one or more motion sensors 280transforms sensed physical movement of the wearable device 122 into asignal indicative of an orientation, a rotation, a velocity, or anacceleration of the wearable device 122. Some embodiments of thewearable device 122 may include an accelerometer but not a gyroscope, ormay include a gyroscope but not an accelerometer.

In some embodiments, the network interface hardware 240 may receiveoperating instructions from the user interface 132. For example, therobot 100 may receive instruction on moving speed and direction of therobot 100 from the user interface 132 and operate the actuator drivehardware 230 based on the received speed and direction. As anotherexample, the robot 100 may receive instructions on operations of the arm106 and operates the arm 106 based on the received instruction.

The robot 100 includes the screen 110 coupled to the communication path206 such that the communication path 206 communicatively couples thescreen 110 to other modules of the robot 100. The screen 110 may displayinformation about identified user that the robot 100 is currentlyinteracting with in response to the identification of the user. Forexample, the screen 110 of the robot 100 may receive a picture of theuser captured by the image capturing devices 102 a, 102 b and displaythe picture of the user. As another example, the screen 110 of the robot100 may display the name of the identified user, which may be retrievedfrom the individual profile for the identified user. In someembodiments, information about a user who remotely operates the robot100 may be output on the screen 110. For example, in FIG. 1B, a liveimage of user C 150 may be transmitted from the remote computing device140 to the robot 100, and the screen 110 of the robot 100 displays thelive image of user C 150. As another example, a static phot or a textualname card of user C 150 may be displayed on the screen 110 of the robot100. The content of identification information may be determined basedon the individual profile for the identified user.

The robot 100 includes the microphone 112 coupled to the communicationpath 206 such that the communication path 206 communicatively couplesthe microphone 112 to other modules of the robot 100. The microphone 112may be configured for receiving user voice commands and/or other inputsto the robot 100. The microphone 112 transforms acoustic vibrationsreceived by the microphone 112 into a speech input signal. As will bedescribed in further detail below, the one or more processors 202 mayprocess the speech input signals received from the microphone 112 toidentify the user from which the acoustic vibrations are originated. Insome embodiments, the robot 100 may not include the microphone 112.

The robot 100 includes the speaker 114 coupled to the communication path206 such that the communication path 206 communicatively couples thespeaker 114 to other modules of the robot 100. The speaker 114transforms data signals into audible mechanical vibrations. The speaker114 outputs audible sound such that a user proximate to the robot 100may interact with the robot 100.

The remote computing device 140 includes a controller 260 that includesone or more processors 262 and one or more memory modules 264, the oneor more image capturing devices 142, network interface hardware 268, thescreen 144, the microphone 146, the user interface 148, and a speaker149. The one or more processors 262, one or more memory modules 264, theone or more image capturing devices 142, network interface hardware 268,the screen 144, the microphone 146, and the user interface 148 may becomponents similar to the one or more processors 202, one or more memorymodules 204, the image capturing devices 102 a, 102 b, the networkinterface hardware 240, the screen 110, the microphone 112, the speaker114 and the user interface 132 described above.

Referring now to FIG. 3, a flowchart of a method 300 of implementing atask requested by a user is schematically depicted. In some embodiments,the method 300 may be implemented as logic within the machine readableinstructions that, when executed by the one or more processors 202,obtain information about a user (e.g., a person, or another robot)proximate to the robot with the one or more sensors, identify the userbased on the obtained information, obtain an action of the user,retrieve an individual profile for the user based on the identifieduser, determine an intended instruction related to a task based on theaction of the user related to the task and the individual profile forthe user, and implement the task based on the intended instruction. Itis noted that, while the method 300 depicts a specific sequence,additional embodiments of the present disclosure are not limited to anyparticular sequence.

Referring now to FIG. 3, at block 310, the robot 100 obtains informationabout a user proximate to the robot 100 with one or more sensors. One ormore sensors may include the image capturing devices 102 a, 102 b,and/or the microphone 112. For example, the robot 100 may receive imagedata representative of the operating space 101 including user A 120 oruser B 130 as shown in FIG. 1A. As noted above, in some embodiments, therobot 100 operates within the operating space 101 and is configured toacquire image data of the operating space 101 from the image capturingdevices 102 a, 102 b, and to then send the acquired image data of theoperating space 101 to the one or more processors 202 and/or the one ormore memory modules 204 for storage and/or processing. In someembodiments, the robot 100 may receive image data from a source externalto the robot 100 (e.g., the mobile device 250 or the remote computingdevice 140), such as via the network interface hardware 240.

The image data received at block 310 may be data of a variety of forms,such as, but not limited to red-green-blue (“RGB”) data, depth imagedata, three dimensional (“3D”) point data, and the like. In someembodiments, the robot 100 may receive depth image data from an infraredsensor or other depth sensor, such as an infrared sensor or depth sensorintegrated with the image capturing devices 102 a, 102 b. In otherembodiments that include a depth sensor (e.g., an infrared sensor), thedepth sensor may be separate from the image capturing devices 102 a, 102b.

In some embodiments, the robot 100 may receive audio data from user A120 or user B 130. As noted above, in some embodiments, the robot 100operates within an operating space 101 and is configured to acquire theaudio data from the microphone 112, and to then send the acquired audiodata of the operating space 101 to the one or more processors 202 and/orthe one or more memory modules 204 for storage and/or processing.

In some embodiments, the robot 100 may receive information about a userproximate to the robot 100 from the mobile device 250 or the remotecomputing device 140. For example, a user may log in to the robot 100using the mobile device 250. Specifically, a username and a password aretransmitted from the mobile device 250 or the remote computing device140 to the controller 210 via the network interface hardware 240.

Still referring to FIG. 3, at block 320, the robot 100 identifies theuser based on the obtained information. In embodiments, the one or moreprocessors 202 may implement face recognition algorithms on the capturedimages to identify the user. One or more face algorithm modules may bestored in the one or more memory modules 204, and implanted by the oneor more processors 202. Any known face recognition algorithms may beused to identify the user.

In some embodiments, the robot 100 identifies the user based on theacquired audio data from the microphone 112. For example, the one ormore processors 202 receive the acquired audio data from the microphone112 and compare the acquired data with samples stored in the one or morememory modules 204. The one or more processors 202 may extract thefeatures of the acquired audio data, e.g., a frequency, an amplitude, anintonation, etc., and compare the extracted features with the featuresof samples to identify the user who generated the audio.

In some embodiments, the robot 100 identifies the user based on theinformation received from the mobile device 250 or the remote computingdevice 140. For example, in FIG. 1B, a user types a username and apassword into the remote computing device 140. The remote computingdevice 140 transmits the information about the user name and password tothe robot 100. The robot 100 may identify the user based on the receivedusername and password.

In some embodiments, the screen 110 of the robot 100 may displayinformation about the identified user that the robot 100 is currentlyinteracting with in response to the identification of the user. Forexample, the screen 110 of the robot 100 may display the picture of theidentified user. As another example, the screen 110 of the robot 100 maydisplay the name of the identified user, which may be retrieved from theindividual profile for the identified user.

Still referring to FIG. 3, at block 330, the robot 100 obtains an actionof the user. In embodiments, the robot 100 may capture the movements ofthe user using the image capturing devices 102 a and 102 b. For example,the robot 100 may capture videos of the user and analyze the video toobtain the action of the user. Specifically, the one or more processors202 may receive the captured video and process the video to identify theaction of the user, e.g., shaking hands, shaking a head, pointing anobject with an index finger, lowering or raising a hand, etc. One ormore action recognition modules may be stored in the one or more memorymodules 204, and implanted by the one or more processors 202.

In some embodiments, the robot 100 may receive motion data from thewearable device 122 worn by a user. For example, the robot 100 mayreceive motion information about an orientation, a rotation, a velocity,and/or an acceleration of the wearable device 122 sensed by the one ormore motion sensors 280. In some embodiments, the robot 100 may receiveoperation instructions from the user interface 132. For example, therobot 100 may receive data related to the movement of a joystick (e.g.,an orientation and/or tilt angle of the joystick) of the user interface132.

Still referring to FIG. 3, at block 340, the robot 100 retrieves anindividual profile for the user based on the identified user. Inembodiments, the one or more memory modules 204 may store individualprofiles for various users. The various users may be pre-registered tothe robot 100, and individual profiles for the pre-registered users maybe stored in the one or more memory modules 204 of the robot 100. Eachof the individual profiles may store various individualized data for theusers. For example, each of the individual profiles may store gains andparameters that adjust the maximum speed of the movement of the robot100, or maximum torque or force that the robot 100 may apply.

In some embodiments, each of the individual profiles may store workspacescaling values. The workspace scaling values represent a ratio betweenthe movement of a user and the movement of the robot. For example, anindividual profile for user A stores a workspace scaling valueindicating that a ratio of user's movement to the robot's movement is 5to 1 (i.e., a user moves her arm 5 inches and the robot moves its arm 1inch in response to the user's movement). An individual profile for userB stores a workspace scaling value indicating that a ratio of user'smovement to the robot's movement is 1 to 5 (i.e., a user moves her arm 1inch in a certain direction and the robot moves its arm 5 inches in thedirection in response to the user's movement).

In some embodiments each of the individual profiles may store apreferred view. For example, a profile for user A may indicate that theuser A prefers an overhead view of the robot 100 when the user Aremotely operates the robot 100. As another example, a profile for userB may indicate that the user B prefers to viewing from the robot'sperspective when the user B remotely operates the robot 100.

In some embodiments each of the individual profiles may store apreferred feedback format. For example, a profile for user A mayindicate that the user A prefers a slight haptic feedback through ajoystick when the robot that the user A is interacting with bumps intoan obstacle. As another example, a profile for user B may indicate thatthe user B prefers an audible feedback through a speaker when the robotthat the user B is interacting with bumps into an obstacle.

In some embodiments, individual profiles may store user intentparameters. For example, a profile for user A may indicate that the userA requires precise movements, whereas a profile for user B may indicatethat the user B does not require precise movements. As such, when arobot is working for user A, the robot operates accurately (e.g.,placing a cup at the right location designated by the user A). When arobot is working for user B, the robot compromises accuracy of movement,but operates faster (e.g., it does not spend much time in order to placea cup at the exact position).

In some embodiments, the individual profiles may store actions inassociation with user's intentions. For example, a profile for user Amay store an action of thumbs-down as related to disapproval, and aprofile for user B may store an action of shaking her head as related todisapproval. When a robot is working for user A and user A makes athumbs-down gesture, the robot interprets that user A does not approvethe action that the robot is going to initiate. Similarly, when a robotis working for user B and user B shakes her head, the robot interpretsthat the user B does not approve the action that the robot is going toinitiate.

In some embodiments, the gains, parameters, workspace scaling values,and other values may be manually set and/or adjusted by users. Forexample, a user may operate the robot 100 by moving his arm wearing thewearable device 122 or manipulating the joystick of the user interface132, and adjust the sensitivity of movement of the robot 100 byadjusting the gains, parameters, and/or workspace scaling values foroperating the robot 100.

In some embodiments, the individual profiles may be stored remotely, andthe robot 100 may receive the stored individual profiles from anexternal device through the network interface hardware 240.

Still referring to FIG. 3, at block 350, the robot 100 determines anintended instruction related to a task based on the action of the userrelated to the task and the individual profile for the user. Forexample, the robot 100 may identify a user proximate to the robot 100 asuser C and obtain user C's action of pointing an index finger at abottle by processing images captured by the image capturing devices 102a and 102 b. The individual profile for user C stores interpretation ofgestures of user C, for example, interpreting the action of pointing theindex finger at an object as an instruction for moving toward the user.The individual profile for user C may also include the speed of therobot, e.g., 0.1 meter/second. Based on the user C's pointing with theindex finger and the individual profile for user C, the robot 100determines that user C is instructing the robot 100 to move to thebottle at the speed of 0.1 meter/second. As another example, the robot100 may identify a user proximate to the robot 100 as user D and obtainuser D's action of pointing an index finger at a bottle by processingimages from the image capturing devices 102 a and 102 b. The individualprofile for user D stores interpretation of gestures of user D, forexample, interpreting the action of pointing the index finger at a useras an instruction for picking up the user. The individual profile foruser D may also include the speed of the robot, e.g., 0.2 meter/second.Based on the user D's pointing with the index finger and the individualprofile for user D, the robot 100 determines that user D is instructingthe robot 100 to move toward the bottle at the speed of 0.2 meter/secondand pick up the bottle.

In some embodiments, the screen 110 of the robot 100 may displayinformation about the task that the robot 100 is currently implementingin response to determining the intended instruction. For example, thescreen 110 of the robot 100 may display text describing the task thatthe robot 100 is currently implementing, e.g., “Picking up a waterbottle.” In some embodiments, the display of the mobile device 250 orthe remote computing device 140 may display information about the taskthe robot 100 is currently implanting.

Still referring to FIG. 3, at block 360, the robot 100 implements thetask based on the intended instruction. For example, in response to thedetermination that user C is instructing the robot 100 to move to thebottle at the speed of 0.1 meter/second, the robot 100 moves toward thebottle at the speed of 0.1 meter/second. As another example, in responseto the determination that user D is instructing the robot 100 to pick upthe bottle, the robot 100 moves toward the bottle at the speed of 0.2meter/second and picks up the bottle.

FIGS. 4A and 4B depict operations of the robot 100 based on individualprofiles including workspace scaling values, according to one or moreembodiments shown and described herein. In FIG. 4A, the robot 100identifies the user 410 and retrieves the individual profile for theuser 410. The individual profile for user 410 may include a workspacescaling value of 1 to 2. That is, the ratio of the movement of the user410 and the movement of the robot 100 is 1 to 2. The wearable device 122on the arm of the user 410 monitors the movement of the arm andtransmits the movement information to the robot 100. For example, whenthe user 410 lowers his arm by one inch, the wearable device 122transmits the dislocation information (e.g., moving 1 inch toward—ydirection) to the robot 100. Based on the dislocation of the arm of theuser 410 and the individual profile for user 410, the robot 100 movesits arm in—y direction by two inches.

In some embodiments, the user 410 may not wear the wearable device 122.The robot 100 may capture images of the user 410 with the imagecapturing devices 102 a and 102 b, and process the images to determinethe dislocation of the arm of the user 410. Based on the determineddislocation of the arm of the user 410 and the individual profile foruser 410, the robot 100 moves its arm in—y direction by two inches.

In FIG. 4B, the robot 100 identifies the user 420 and retrieves theindividual profile for the user 420. The individual profile for user 420may include a workspace scaling value of 2 to 1. That is, the ratio ofthe movement of the user 420 and the movement of the robot 100 is 2to 1. The wearable device 122 on the arm of the user 420 monitors themovement of the arm and transmits the movement information to the robot100. For example, when the user 420 lowers his arm by two inches, thewearable device 122 transmits the dislocation information (e.g., moving2 inches toward—y direction) to the robot 100. Based on the dislocationof the arm of the user 420 and the individual profile for user 420, therobot 100 moves its arm in—y direction by one inch. In some embodiments,the user 420 may not wear the wearable device 122. The robot 100 maycapture images of the user 420 by the image capturing devices 102 a and102 b, and process the images to determine the dislocation of the arm ofthe user 420.

FIGS. 5A and 5B depict operating of the robot based on individualprofiles, according to embodiments shown and described herein. In FIG.5A, the robot 100 is holding an object 540 (e.g., a water bottle). Therobot 100 identifies the user 510 and obtains the action of the user510, e.g., the action of pointing at a point 532 on a table 530 with twofingers. The robot 100 retrieves the individual profile for the user510. The individual profile for the user 510 may include mapping theaction of pointing with two fingers to an instruction for putting downan object at the pointed location. The individual profile for the user510 also includes a maximum allowance deviation from the pointedlocation, e.g., 10 inches from the point 532. The area 534 illustratesan area where the robot 100 is allowed to put down the object 540 basedon the maximum allowance deviation. Based on the action of pointing atthe point 532 with two fingers and the individual profile for user 510,the robot 100 moves toward the table 530 and puts the object 540 anylocation within the area 534.

In FIG. 5B, the robot 100 is holding an object 540 (e.g., a waterbottle). The robot 100 identifies the user 520 and obtains the action ofthe user 520, e.g., the action of pointing at a point 532 on the table530 with two fingers. The robot 100 retrieves the individual profile forthe user 520. The individual profile for the user 520 may includemapping an action of pointing with two fingers to an instruction forputting down an object at the pointed location. The individual profilefor the user 520 also includes a maximum allowance deviation from thepointed location, e.g., 0.5 inches from the point 532. The area 536illustrates an area where the robot 100 is allowed to put down theobject 540. Based on the action of pointing at the point 532 with twofingers and the individual profile for user 520, the robot 100 movestoward the table 530 and puts the object 540 within the area 536. Assuch, when the robot 100 is interacting with user 520, the robot 100 mayimplement the task with more precision (e.g., putting an object at theright position) compared to when interacting with the user 510. Incontrast with the user 520, when the robot is interacting with the user510, the robot 100 may complete the task more quickly than wheninteracting with the user 520 because the individual profile for theuser 510 allows greater deviation.

It should now be understood that obtaining, by one or more sensors of aremote computing device, information about a user proximate to theremote computing device; identifying, by a controller of the remotecomputing device, the user based on the obtained information; obtaining,by the controller, an action of the user; retrieving, by the controller,an individual profile for the user based on the identified user;determining, by the controller of the remote computing device, anintended instruction related to a task based on the action of the userrelated to the task and the individual profile for the user; andinstructing, by the controller of the remote computing device, the robotto implement the task based on the intended instruction, may provide forcustomized task implementation for different users. Adjusting operatinggains, parameters, and/or workspace scaling values of a robot systemaccording to individual profiles may provide for enhanced interactionexperience with robot systems. Furthermore, different maximum allowancedeviations for a certain task may also enhance human-robot interactionexperiences.

While particular embodiments have been illustrated and described herein,it should be understood that various other changes and modifications maybe made without departing from the spirit and scope of the claimedsubject matter. Moreover, although various aspects of the claimedsubject matter have been described herein, such aspects need not beutilized in combination. It is therefore intended that the appendedclaims cover all such changes and modifications that are within thescope of the claimed subject matter.

What is claimed is:
 1. A teleoperation system comprising: a robotcomprising an actuator configured to move at least a portion of therobot; and a remote computing device comprising: one or more processors;one or more sensors communicatively coupled to the one or moreprocessors; a non-transitory memory component communicatively coupled tothe one or more processors; and machine readable instructions stored inthe non-transitory memory component that cause the remote computingdevice to perform at least the following when executed by the one ormore processors: obtain information about a user proximate to the remotecomputing device with the one or more sensors; identify the user basedon the obtained information; obtain an action of a user; retrieve anindividual profile for the user based on the identified user; determinean intended instruction related to a task based on the action of theuser related to the task and the individual profile for the user; andinstruct the robot to implement the task with the actuator based on theintended instruction, wherein the individual profile includes intentparameters related to the action of the user.
 2. The teleoperationsystem of claim 1, wherein the individual profile includes a workspacescaling value.
 3. The teleoperation system of claim 2, wherein the robotfurther comprises an arm, wherein the machine readable instructionsstored in the non-transitory memory component cause the robot to operatethe arm with the actuator based on the action of the user and theworkspace scaling value.
 4. The teleoperation system of claim 1, whereinthe individual profile includes one or more operating gains of therobot.
 5. The teleoperation system of claim 1, wherein the one or moresensors includes one or more imaging devices configured to obtain one ormore images of the user.
 6. The teleoperation system of claim 1,wherein: the remote computing device comprises an output deviceconfigured to provide feedback to the user based on an operation of therobot, and the machine readable instructions stored in thenon-transitory memory component cause the remote computing device todetermine the feedback based on the individual profile for the user. 7.The teleoperation system of claim 1, wherein: the remote computingdevice comprises a display configured to display a view related to therobot, and the machine readable instructions stored in thenon-transitory memory component cause the remote computing device todetermine a type of the view based on the individual profile for theuser.
 8. The teleoperation system of claim 1, wherein the individualprofile includes a maximum allowable deviation for implementing thetask.
 9. The teleoperation system of claim 8, wherein the machinereadable instructions stored in the non-transitory memory component,when executed by the one or more processors, cause the remote computingdevice to identify a location related to the task based on the action ofthe user, and implement the task within the maximum allowable deviationfrom the location.
 10. The teleoperation system of claim 1, furthercomprising a screen configured to display information related to theintended instruction.
 11. A robot system comprising: one or moreprocessors; one or more sensors communicatively coupled to the one ormore processors; a non-transitory memory component communicativelycoupled to the one or more processors; and machine readable instructionsstored in the non-transitory memory component that cause the robotsystem to perform at least the following when executed by the one ormore processors: obtain information about a user proximate to the robotsystem with the one or more sensors; identify the user based on theobtained information; obtain an action of a user; retrieve an individualprofile for the user based on the identified user; determine an intendedinstruction related to a task based on the action of the user related tothe task and the individual profile for the user; and implement the taskbased on the intended instruction, wherein the individual profileincludes intent parameters related to the action of the user.
 12. Therobot system of claim 11, wherein the individual profile includes aworkspace scaling value.
 13. The robot system of claim 12, furthercomprising an arm, wherein the machine readable instructions stored inthe non-transitory memory component cause the robot system to operatethe arm based on the action of the user and the workspace scaling value.14. The robot system of claim 12, wherein the machine readableinstructions stored in the non-transitory memory component cause therobot system to receive information on the action of the user from awearable device.
 15. A method for operating a robot, the methodcomprising: obtaining, by one or more sensors of a remote computingdevice, information about a user proximate to the remote computingdevice; identifying, by a controller of the remote computing device, theuser based on the obtained information; obtaining, by the controller ofthe remote computing device, an action of the user; retrieving, by thecontroller of the remote computing device, an individual profile for theuser based on the identified user; determining, by the controller of theremote computing device, an intended instruction related to a task basedon the action of the user related to the task and the individual profilefor the user; and instructing, by the controller of the remote computingdevice, the robot to implement the task based on the intendedinstruction, wherein the individual profile includes intent parametersrelated to the action of the user.
 16. The method of claim 15, whereinthe individual profile includes a workspace scaling value.
 17. Themethod of claim 15, wherein the individual profile includes one or moreoperating gains of the robot.
 18. The method of claim 15, wherein theone or more sensors includes one or more imaging devices configured toobtain one or more images of the user.
 19. The method of claim 15,wherein the individual profile includes a maximum allowable deviationfor implementing the task.
 20. The method of claim 19, furthercomprising: identifying a location related to the task based on theaction of the user; and implementing the task within the maximumallowable deviation from the location.